Research: seven-component aggregation performance stack - #5

Summary

This draft preserves an evidence-backed aggregation performance research stack and now integrates upstream main at f7b433052bceb623222f337556e7d7eca21c6ed1.

It is a research handoff, not a merge-ready production patch. Replay selectors and the measurement seam remain so the retained campaigns are reproducible. The mechanisms should be split into smaller review units after the target architecture and acceptance budget are agreed.

Source and evidence boundary

• The six-component all-off/on campaign binds to b61a0ee64a9e5f41c368f937c56cdbb74fd3908b on the older 84fbd3e freeze.
• The incremental L0/seven-component campaign binds to 256928f9127c812749e40b08b4cf9744185c6b61 on that same freeze.
• The two-kernel campaign binds to performance-bearing commit a5477b369ee44ef7aea91b0799f7b920b349632f on integrated upstream e9cd16d.
• Current PR head 5b7dc20fd6dfef45661e382c0bf1d716fdd174f7 merges upstream f7b4330 without rewriting any evidence-bound commit.

Upstream 53631e8 intentionally replaced the old per-query equality challenge with one power-weight batching challenge per PCS level. The integration retains that complete current protocol across the prover and all verifiers while preserving the research timing spans, direct-fold path and hardened L0 induction selector. Proof hashes, proof sizes, fixtures and performance receipts from before 53631e8 remain historical evidence for their named commits. They are not current-head claims. Wire version 2 identifies the fixture and artifact encoding, not protocol compatibility, so old fixtures and artifacts must be regenerated.

Earlier seven-component result

On the older measured freeze:

• Six components versus all off at N2: outer proving -30.91%, process wall -20.72%, peak physical memory -22.98%.
• Six components versus all off at N8: outer proving -31.87%, process wall -26.85%, peak physical memory -32.44%.
• Incremental L0 NTT at N2: L0 induction -78.02%, PCS opening -25.66%, outer proving -7.07%, process wall -3.51%, peak physical memory -2.48%.
• Incremental L0 NTT at N8: L0 induction -74.10%, PCS opening -24.32%, outer proving -6.51%, process wall -4.46%, peak physical memory -3.27%.
• All 142 retained proofs were byte-identical within topology and passed the unchanged inspection path.

After seven components, the N2 phase cluster was Bus 0.687050 s, PCS opening 0.527055 s, constraints 0.518037 s and Flock reduction 0.336672 s. This moves the previous PCS bottleneck; it does not prove that aggregation meets a production budget.

Two-kernel result on e9cd16d

Two default-off, fail-closed experiments were measured on the earlier integrated stack:

1. LEANVM_CONSTRAINT_NODE_SKIP=1 derives one Boolean constraints-sumcheck endpoint from the running claim, including the exact zeta == 1 branch.
2. FLOCK_PACKED_128_PARALLEL=1 serializes the three live packed witnesses concurrently into disjoint, completely initialized outputs.

The canonical N2 campaign used one AMD EPYC 9354 NUMA domain, CPUs 8 through 15, four pilots and 32 measured fresh processes in a repeated 2 by 2 Williams-square design.

All 36 proofs were byte-identical and passed inspection. The direct mechanism results were repeatedly favorable, but the stated -350 ms system-materiality gate failed. The system deltas are descriptive rather than a confirmatory production-acceptance result.

Current upstream integration

The f7b4330 merge resolved two marked WHIR conflicts and one compile-only batching-state conflict:

• Upstream lambda, power_weights and glue_pending semantics govern the transcript.
• Research Gate0 timing spans and direct-fold6 materialization remain.
• The L0 NTT policy and its fail-closed override remain, using the new arbitrary-weight induction interface.
• The Gate0 diagnostic now reports the effective policy instead of only the inherited heuristic.
• Stale alpha/beta transcript documentation and the handoff evidence boundary were corrected.

Validation on 5b7dc20

The unsealed Apple M4 Pro integration run passed:

• cargo testall.
• cargo test --release --workspace --all-features.
• cargo clippyall.
• cargo fmt --all -- --check.
• cargo docall.
• Ruff formatting and lint checks for python-verifier/verifier.py.
• The complete LaTeX build.
• Both recursion_2to1 variants with LEANVM_PCS_DIRECT_FOLD6=1, with logs confirming direct-fold6 selection on production-shaped openings.
• The ignored recursion_soundness_binds adversarial suite with direct-fold6 forced.

This establishes current-source compatibility and soundness regression coverage. It is not a replacement performance campaign.

Feedback requested

1. What N2 latency, peak-memory, proof-size and concurrency budgets should define acceptance?
2. Is current main the intended architecture for splitting and productionizing these changes?
3. Which mechanism should be reviewed first: direct fold-6, the allocation/capacity group, AVX-512 Bus MLE, the narrow L0 induction policy, the constraints node skip, or packed serialization?